{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "view-in-github",
    "colab_type": "text"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/bradleywhitlock/CDP-Exploration/blob/master/Part_3_Asset_data_dive.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eAXnMj5E9jWG"
   },
   "source": [
    "# Part 3: Asset data dive\n",
    "Let's get started with a guided exploration of the Valhall Platform. In this notebook we will pick one of the equipment that we visualized in operational intelligence and take a closer look at all of the available data!\n",
    "\n",
    "\n",
    "## Quick links\n",
    "* Back to the [Hackathon github repo](https://github.com/cognitedata/open-industrial-data/tree/master/workshops/uni-hackathon)\n",
    "* Documentation of [CDP concepts](https://doc.cognitedata.com/concepts/)\n",
    "* Reference documentation for the [Python SDK](https://cognite-sdk-python.readthedocs-hosted.com/en/latest/))\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "k8-Oj493ReNU"
   },
   "source": [
    "# Step 0: Environment Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4Lf5pnaDBLnc"
   },
   "source": [
    "#### Install the Cognite SDK package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code",
    "id": "pbgkP3p59iiK",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "# if you're working in google colab or similar\n",
    "!pip install -q cognite-sdk \"cognite-logger==0.3.*\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CN5J9FmPBRLk"
   },
   "source": [
    "#### Import the required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code",
    "id": "uFkQ73a44wmU",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "from datetime import datetime, timedelta\n",
    "from datetime import datetime\n",
    "from getpass import getpass\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "from cognite import CogniteClient\n",
    "\n",
    "pd.set_option('display.max_rows', 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Pr0CQECZBWW3"
   },
   "source": [
    "#### Connect to the Cognite Data Platform\n",
    "The SDK client is the entrypoint to all data in CDP, and simply requires the API key that you generated in Part 1.\n",
    "\n",
    "When prompted for your API key, use the key generated by open industrial data as mentioned in the Getting Started steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code",
    "id": "E8CdoRFu9n2t",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "client = CogniteClient(api_key=getpass(\"Open Industrial Data API-KEY: \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VKFPBwYsRjHL"
   },
   "source": [
    "# Step 1: Learn about Organizing Industrial Data\n",
    "\n",
    "The Cognite Data Platform organizes digital information about the physical world. The building blocks of this representation are called *resources*, which you can read up on in detail [here](https://doc.cognitedata.com/concepts/#core-concepts).\n",
    "\n",
    "An important resource to understand is the Asset resource. This is the foundation for organizing industrial data -- time series, work orders, event logs and arbitrary files -- from across complex industrial systems.\n",
    "Assets are linked together with parent-child relationships to build a top-down hierarchical tree, known as \"The Asset Hierarchy\".\n",
    "For example, an Asset Hierarchy could look like this:\n",
    "```\n",
    "  Gas Export Compressor\n",
    "    |- First stage export compressor\n",
    "    |    |- Compressor\n",
    "    |    |- Scrubber\n",
    "    |    |- ...\n",
    "    |- Second stage export compressor\n",
    "    |- ...\n",
    "```\n",
    "Timeseries, events, files and other resources are attached to each Asset.\n",
    "\n",
    "The hierarchical structure can make it easier to find the timeseries data that you're looking for. Though there are [other ways](https://doc.cognitedata.com/concepts/#_3d-models-and-revisions) to do this, we'll focus on using the hierarchy today!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "scrolled": true,
    "id": "OlBU8-O-M-em",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "# download a sample of assets up to a certain depth in the hierarchy\n",
    "df_sample_assets = client.assets.get_assets(limit=1000, depth=4).to_pandas().sort_values('depth')\n",
    "df_sample_assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fXtlvohEM-ep",
    "colab_type": "text"
   },
   "source": [
    "# Step 2: Pick an asset for further investigation\n",
    "For the rest of the workshop, you'll be working with one of the subsystems that you visualized in the [LIVE Operational Intelligence System Overview](https://opint.cogniteapp.com/publicdata/infographics/-LOHKEJPLvt0eRIZu8mE) (see below).\n",
    "Either pick an asset yourself below, or let fate decide ;)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "Vjmz4CgtM-ep",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "import random\n",
    "SYSTEM_OVERVIEW_ASSETS = [\n",
    "    '23-ESDV-92501A',\n",
    "    '23-ESDV-92501B',\n",
    "    '23-HA-9103',\n",
    "    '23-PV-92538',\n",
    "    '23-VG-9101',\n",
    "    '23-KA-9101',\n",
    "    '23-HA-9115',\n",
    "    '23-HA-9114',\n",
    "    '23-FV-92543',\n",
    "    '23-ESDV-92551A',\n",
    "    '23-ESDV-92551B',\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "THfDWvRgM-er",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "# fetch the asset metadata from CDP using the assets client\n",
    "\n",
    "df_system_overview_assets = pd.concat([\n",
    "    client.assets.get_assets(name=n).to_pandas()\n",
    "    for n in SYSTEM_OVERVIEW_ASSETS\n",
    "])[['name', 'id', 'parentId', 'description']].set_index('name')\n",
    "\n",
    "df_system_overview_assets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "nVuSQ4eQM-et",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "# Choose an asset for analysis, or let fate decide :)\n",
    "\n",
    "asset_name = random.choice(SYSTEM_OVERVIEW_ASSETS)\n",
    "\n",
    "asset_id = df_system_overview_assets.loc[asset_name, 'id']\n",
    "\n",
    "print(\"And my asset is!\")\n",
    "df_system_overview_assets.loc[asset_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o6Eeyrp2M-ev",
    "colab_type": "text"
   },
   "source": [
    "# Step 3: Find all the timeseries for your asset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X75GOITO_m1q"
   },
   "source": [
    "The interface `client.assets.get_asset_subtree()` can be used to retrieve all of the *children* of an Asset. The `depth` parameter sets how far we traverse down the hierarchy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab_type": "code",
    "id": "1OPecRPv6xsU",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "df_asset_children = client.assets.get_asset_subtree(\n",
    "    asset_id=df_system_overview_assets.loc[asset_name, 'id'],\n",
    "    depth=10\n",
    ").to_pandas().sort_values('depth')\n",
    "df_asset_children[['depth', 'id', 'parentId', 'description']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "56ep933oM-e2",
    "colab_type": "text"
   },
   "source": [
    "... Assets are interesting to see how things are put together, but what I'm sure you're really after are those petabytes of **time series**; those beautiful pressure (PT), temperature (TT) and flow (TT) sensors that have recorded the life of the platform for the last few years.\n",
    "\n",
    "First we need to find all these time series. We can use the `path` parameter in the `time_series` client to get all the time series attached to assets below our system overview asset. Note that this parameter maps directly to the CDP API, and therefore needs to provide the asset id formatted carefully as a json string: `\"[id, ]\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "PDGvhvkPM-e3",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "df_asset_children_timeseries = client.time_series.get_time_series(path=str([asset_id])).to_pandas()\n",
    "df_asset_children_timeseries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jGqKrad1M-e5",
    "colab_type": "text"
   },
   "source": [
    "Great! We have discovered the timeseries below our asset!\n",
    "\n",
    "**Note**: CDP can also store string and step timeseries. Step timeseries have different aggregation methods, and support dead-band-compression for time series that do not change very often (e.g. valve opening angles)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LXxQ2BJDM-e5",
    "colab_type": "text"
   },
   "source": [
    "# Step 4: Into the timeseries datapoints!\n",
    "In CDP we do some very clever things in the backend to store serve up timeseries just the way you like it:\n",
    "- Store the timeseries (timestamp, value) in their raw format, because one day we'll need it\n",
    "- Precompute aggregations for millisecond response times\n",
    "- Build tabular structures server side\n",
    "- Enable natural language time specifications (e.g. `start='8d-ago'` and `granularity='10m'`)\n",
    "\n",
    "So once you've located the timeseries you're interested in analyzing, the `datapoints` client has several options for downloading the data.\n",
    "\n",
    "**Note:** The timeseries column is represented throughout CDP as milliseconds since epoch time. Pandas offers an easy conversion to python datetime with `pd.to_datetime(<column/value>, unit='ms')`.\n",
    "\n",
    "**Note** The *get_datapoints_frame()* SDK function can accept maximum of 100 time series, this is why we have to make multiple calls then concatenate the Data Frames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "AHiqEG4bM-e6",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "# set aside string time series for now because they do not aggregate together with numerical time series\n",
    "# consider investigating the string timeseries in part 4b\n",
    "lst_timeseries = df_asset_children_timeseries[~df_asset_children_timeseries['isString']]['name'].tolist()\n",
    "lst_timeseries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "KY13QJszM-e8",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "# we can request up to 100 time series because of api limitations, so let's request this data batch-by-batch\n",
    "df_data = None\n",
    "for start_idx in range(0, len(lst_timeseries), 100):\n",
    "    df_data_batch = client.datapoints.get_datapoints_frame(\n",
    "        time_series=lst_timeseries[start_idx: start_idx+100],\n",
    "        aggregates=['avg'],\n",
    "        granularity='1h',\n",
    "        start='30d-ago',\n",
    "    )\n",
    "    \n",
    "    if df_data is None:\n",
    "        df_data = df_data_batch.copy()\n",
    "        df_data = df_data.set_index(pd.to_datetime(df_data['timestamp'], unit='ms')).drop('timestamp', axis=1)\n",
    "    else:\n",
    "        df_data_batch = df_data_batch.set_index(pd.to_datetime(df_data_batch['timestamp'], unit='ms')).drop('timestamp', axis=1)\n",
    "        df_data = pd.concat([df_data, df_data_batch], axis=1)\n",
    "    \n",
    "df_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "id": "FGe2C20eM-e_",
    "colab_type": "code",
    "colab": {}
   },
   "outputs": [],
   "source": [
    "# plot up to 10 random rows\n",
    "df_plot_sample = df_data[random.sample(list(df_data.columns), min(10, len(df_data.columns)))]\n",
    "\n",
    "df_plot_sample.plot(subplots=True, figsize=(20,20));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d21kj8CWNpNN"
   },
   "source": [
    "# Congratulations, you are done with part 3!\n",
    "\n",
    "Save your notebook, and remember your asset for the next part, where we dig deeper into the data."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "Part 3 - Asset data dive.ipynb",
   "version": "0.3.2",
   "provenance": [],
   "collapsed_sections": [],
   "include_colab_link": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
